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PREFACE 


"studied  on  Bottleneck  Problems  in  Production  Processes' 
consists  of  two  papersy^  their  purpose  being  to  study  In  detail 
some  representative  problems  which  occur  In  the  study  of  Inter¬ 
industry  processes.  The  general  techniques  will  be  those  of 
the  theory  of  dynamic  programming.  A  brief  description  of  these 
techniques  appears  In  [[l]  ;  a  longel  expository  account,  together 
with  aux  app^ldwat^lon;  appears  In  QP]  . 

'^The  first  paper,  by  R.  Sherman  Lehman,  considers  a  simple 
mathematical  model  of  the  Interaction  of  two  Interdependent 
Industries  engaged  In  the  production  of  one  Item.  For  conveni¬ 
ence  ,  Hse-et^e^^—eaU—theae-  the  auto  and  steel  Indus  trie  s^lc- ^ 
Utilizing  a  highly  lumped  model,  we  shall  assume  that  at 
any  time  t  the  state  of  the  Industries  Is  completely  specified 
by  the  following  quantities: 


(1) 


Xi(t)  >  auto  stockpile  at  time  t 
XaCt)  >  auto  capacity  at  tine  t 
X3(t)  ■  steel  stockpile  at  tine  t 
X4(t)  *  steel  capacity  at  time  t 


n 


1 1< 
/A' 


Taking  t  to  be  a  continuous  variable,  at  each  Instant  we 
must  determine  rates  of  allocation  of  the  steel  stockpile  towards 
three  distinct  objectives: 
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(a) 

Production 

of  autos 

(2) 

(b) 

Building  of 
capacity 

auto  factories,  l.e.. 

Increase  of  auto 

(c) 

Building  of 
capacity 

steel  factories;  l.e. 

,  Increase  of  steel 

The  last  two  of  these  three  objectives  are  to  be  sublimated 
to  tie  .wer-all  objective  of  maximizing  the  total  number  of  autos 
produced  over  a  time-period  T,  which  Is  to  say,  the  quantity  Xi(T). 

"'he  measures  of  stockpile  and  capacity  are  chosen  so  that 
one  unit  of  capacity,  either  auto  or  steel,  Is  required  for  the 
production  of  one  unit  of  stocxpile  in  unit  time.  We  assume 
that  bi  units  of  steel  are  required  to  make  one  unit  of  autos, 
bs  units  of  steel  to  increase  auto  capacity  by  one  unit,  and  b4 
units  of  steel  to  increase  steel  capacity  by  one  unit.  However, 
we  shall  assume  that  no  steel  is  required  to  produce  additional 
steel . 

Finally,  we  shall  assume  that  there  are  no  time-lags  between 
allocation  and  production,  an  assumption  of  some  magnitude. 

Let 


(^) 


(a)  Zi(t)  =  rate  of  production  of  autos 

(b)  Z2(t)  =  rate  of  Increase  of  auto  capacity 

(c)  Z3(t)  -  rate  of  production  of  steel 

(d)  Z4(t)  =  rate  of  Increase  of  steel  capacity 


We  obtain  the  following  system  of  equations: 


I 
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-  Zi(t),  Xi(0)  ■  Cl, 

■*2(t),  X2(0)-Ci, 

(M 

“  23(t)  —  biZi(t)  —  b2Z2(^)  ~  b4X4(t),  X3(0)  O3, 

.  Z4(t),  X4(0)  -  C4,  . 

together  with  the  linear  constraints 

(a)  z, (t)  <  X2(t) 

(5)  (b)  zs(t)  ^  X4(t) 

(c)  Zj(t)  >  0,  1-1, 2, 5, 4 

(<J)  X3(t)  >  0. 


The  first  two  of  these  constraints  arise  from  the  capacity 
limitations.  The  third  Is  a  statement  that  the  rates  of  pro¬ 
duction  must  be  non-negative,  and  the  fourth  that  the  steel 
stockpile  must  be  non-negative. 


^  The  problem  Is  now  to  determine  the  Zj^(t),  satisfying  the 
reXtrlctlons  of  (5)  which  maximize  Xi(T). 


The  second  papen  by  Richard  Bellman  and  R.  Sherman  Lehman, 
treats  the  problem  of  maximizing  x^(t)  given  the  equations 

biSii‘,^t)  -  4V(t),  Xs(O)  -  c,V 


aaV(b)  -  »i(t). 


x^(0)  <  oa^  ^ 


I 
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and  the  constraints 


3 


'">(«)  2.Z,  *3  fit  0  j 

(b) 


*3" 


{, 


(c)  |«zs  '*’ 


(v-.,  /  H  o, 

--■*  ,  (  l~  ^  ^ 

/ 


(V  t. 


J* 


where  aa,  b*,  y*,  ^3’*  are  positive  constants. 


ti^ 

\ 


of  natheaatical 


This  problem  Is  derived  from  the  aa 
model  as  the  first. 

Both  problems  require  for  their  treatment  an  Intimate  Inter¬ 
mingling  of  general  theory  and  detailed  analysis.  It  seems 
f*i**ly  c***taln  that  It  Is  not  altogether  the  fault  of  our  methods 
that  the  analysis  Is  so  complicated,  but  that  these  are  actually 
complicated  problems  with  nonobvlous  solutions. 
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STUDIES  ON  BOTTLENECK  PROBLEMS  IN  PRODUCTION  PROCESSES 


STUDY  I; 

A  SIMPLE  TWO-INDUSTRY  PROBLEM 
INVOLVING  MULTI-STAGE  PRODUCTION  PROCESSES 

by 

R.  Sherman  Lehman 


§1 .  Introduction 

In  the  Preface  to  these  Studies,  we  have  formulated  in 
mathematical  tems  the  problem  of  utilizing  the  steel  and  auto 
industries  so  as  to  maximize  auto  production.  Let  us  then  con¬ 
tinue  from  equations  (4)  and  (5)  of  the  Preface. 

These  equations  can  be  combined  to  provide  an  equivalent 
system  of  integral  inequalities: 


zi  <X2i  Zi (t)  -  z,(s)ds  ^  ca 

o 

(1.1)  0  ^  xa:  /  (-*3(8)  biZi(B)  -f  baZaCs)  b4Z4(8))d8  ^  C3 

o 

*3  i  Z4:  Z3(t)  -  i  Z4(8)dS  ^  84 


Our  problem  is  a  special  case  of  the  following  more  general 

problem.  Let  Z  be  the  set  of  all  vector  functions  z(t)  which 

# 

satisfy  the  conditions 


(1.2) 


z(t)  ^  0 

t 

Bz(t)  ♦  J*  Cz(8)d8  ^  C 
O 


*  If  u  and  V  are  vectors,  by  the  notation  u  ^  v  we  mean  that  each 
component  of  u  ^  the  corresponding  component  of  v,  cf.  (2]. 
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where  B  and  C  are  matrices,  and  e  Is  a  constant  vector.  The 
problem  Is  to  find  a  vector  function  z(t)  In  Z  which  maximizes 


(1.3) 


J 


(z(t),o)dt. 


the  Integral  of  the  Inner  product  of  z  with  a  constant  vector  a. 

In  [23  •  It  was  shown  that  there  Is  a  dual  problem  which  pro¬ 
vides  a  sufficient  condition  that  a  z(t)  belonging  to  Z  be  a 
maximizing  vector  function,  or  In  other  words,  that  a  feasible 
solution  be  optimal.  Let  W  be  the  set  of  vector  functions  w(t) 
for  which 


(l-M 


w(t)  ^  0 
B'w(t)  ♦  C* 


w(8)ds  >  a 


where  B*  and  C  are  the  transposes  of  B  and  C. 


Is  to  find  Min 
waw 


/ 


(w(t),c)dt.  It  can  be  proved 


The  dual  problem 
readily  that  for 


(1.5) 


j 


T 

(z(t),a)dt  ^  f 


(w(t) ,c)dt. 


and  that  If  we  find  two  vector  functions  for  which  (1.3)  holds 
with  equality,  they  must  yield  the  maxlsnam  and  minimum,  respec¬ 
tively,  for  the  two  problems  (see  [2]).  Two  such  vector  func¬ 
tions  for  which  equality  holds  will  be  said  to  be  paired  with 
each  other.  Thus,  a  sufficient  condition  that  a  z  belonging  to 
Z  be  optimal  Is  that  It  can  be  paired  with  some  w  In  W. 
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For  the  auto-eteel  problem  formulated  above  we  have 


/ 

/I  0  0  0 

1 

lo  -1  0  0  ^ 

h\ 

(1.6)  B-  0000  C- 

bi  ba  —1  b4 

a-  0 

0 

0 

0 

0 

7 

0 

0 

0^ 

V 

The  dual  system  of  Inequalities  Is  therefore 

T 

ii  =  Wa(t)  *  ^  W3(a)d8  -1^0 

T  T 

(1*7)  is  =  *»*(8)ds  *  ba  ^  W3(8)ds  ^  0 

T 

I3  E  W4(t)  -  {  WaCs)  ds  ^  0 
T  T 

^4  =  b4  X  W3(s)ds  -  X  W4(8)d8  >  0. 


We  have  chosen  to  call  the  components  of  w,  Wg ,  wg,  and  W4  In 
order  to  keep  the  connection  with  the  Inequalities  zt  ^  x», 
0^X3,  Z3  1  X4  clear. 

The  optimality  conditions,  l.e.,  the  conditions  that  (1.5) 
hold  with  equality,  are: 


If  z^(t)  >  0,  then  |j(t)  -  0,  (1-1,2, 3, 4) 

If  Zi(t)  <  xa(t),  then  w*(t)  •  0 
If  0  <  Xs(t),  then  Ws(t)  -  0 
If  Z3(t)  <  X4(t),  then  W4(t)  -  0 


(1.8) 
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The  following  are  equivalent  to  the  optimality  conditions: 


If  Ij^(t)  >  0,  then  *j(t)  -  0,  (1-1,8,3,*») 
If  wa(t)  >  0,  then  Z|(t)  -  x*(t) 

If  WaCt)  >  0,  then  0  -  x^it) 

If  W4(t)  >  0,  then  ZaCt)  -  X4(t) 


§2.  Delta-functions 

g^foxv  we  proceed  to  determine  the  solution,  let  us  discuss 
the  use  that  we  will  make  of  delta  functions.  It  can  easily  hap¬ 
pen  that  the  general  problems  discussed  above  have  no  solutions 
If  the  sets  Z  and  W  are  composed  only  of  vectors  having  compo¬ 
nents  which  are  measurable  functions.  In  fact,  as  we  shall  later 
see,  this  Is  the  usual  case  In  the  auto-ateel  problem.  This 
difficulty  can  be  evaded  by  enlarging  the  sets  Z  and  W  so  that 
they  contain  vector  "functions"  whose  components  are  sums  of 
measurable  functions  and  "delta  functions."  In  these  enlarged 
classes  the  problems  have  solutions.  By  a  delta  function  concen¬ 
trated  at  to  with  weight  UX  which  se  denote  by  «C(t-tQ),  we  mean 
an  Improper  function  such  that 


t 

/  6(s-tj,)^(8)ds 

o 


0  If  t  <  tg 
**(to)  If  t  >  t^ 


for  every  function  ^  continuous  at  t^.  (For  t  -  t^  the  Integral 
Is  undefined  except  when  ^(t^)  •  0,  In  which  ease  It  Is  defined 
to  be  0. ) 
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The  use  of  delta  functions  can  be  Justified  rigorously 

either  by  the  alternative  use  of  Stleltjes  Integrals,  or  by 

regarding  the  delta  functions  as  obtained  by  completing  the 

space  of  measurable  functions  by  a  process  similar  to  that  used 

In  obtaining  the  real  numbers  from  the  ratlonals. 

The  optimality  conditions  remain  the  same  even  when  Z  and  W 

are  enlarged  In  the  above  way.  We  observe  that  there  Is  no  harm 

In  the  violation  of  the  optimality  conditions  at  Isolated  points 

or  even  In  sets  of  measure  zero  when  only  measurable  functions 

% 

are  allowed  as  components  of  z  and  w.  But,  when  one  of  the  vec¬ 
tors,  w,  for  example,  has  a  component  Wj^  which  Is  a  delta  function 
at  the  point  t^,  then  for  a  z  to  be  paired"  with  w,  the  corres¬ 
ponding  optimality  conditions  must  be  satisfied  at  the  point  t^. 

We  shall  find  that  we  never  have  to  use  delta  functions  con¬ 
centrated  at  any  point  other  than  0  to  obtain  an  optimal  z.  Intui¬ 
tively,  this  means  that  radical  changes  are  not  necessary  except 
at  the  beginning. 

^5.  The  Solution 

The  procedure  that  we  use  will  be  to  construct  a  number  of 
w-aolutlons  which  we  can  pair  with  z's  belonging  to  Z  and  hence 
obtain  solutions  of  our  problem.  The  chief  difficulty  occurs  In 
constructing  w-aolutlons  with  suitable  properties.  In  this  we 
are  guided  by  a  combination  of  guessing  and  observation  of  proper¬ 
ties  that  an  optimal  z  should  have. 

*  Cf.  0.  Temple,  "Theorels  and  Applications  of  Generalized  Func¬ 
tions,"  Jour.  Lond.  Math.  Soc.,  28  (1953).  PP*  13*^— iW. 
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First  of  all.  It  Is  clear  that  we  should  always  have 

Z3  -  X4.  To  produce  too  much  steel  Is  not  harmful.  This  tells 

T 

us  that  we  should  have  isCt)  -  0  for  all  t;  l.e.,  W4(t)  -  ^  W3(s)ds. 
The  remaining  Inequalities  of  (1.7)  then  beeosw 


(3.1) 


li  =  w*(t)  -f 


Je  =  baW4(t) 


i4  =  b4W4(t) 


-J 

t 

-/ 

t 


t)  -  1  ^  0 
W2(s)ds  i  0 
W4(s)ds  ^  0 


Shortly  before  T  It  Is  clear  that  we  should  be  producing  Xi 
since  that  Is  the  quantity  we  wish  to  maximize  at  time  T.  Hence, 
we  will  have  Zj  >  0,  which  Implies  that  it  -  0.  This  alone  will 
not  give  us  sufficient  Information  to  determine  w^  and  w«. 

Ve  first  construct  a  w  solution,  which  we  shall  call  the 
basic  w-solutlon,  with  the  property  that  It  -  0  near  the  end. 
This  means  that  we  must  have  W4(T)  -  0.  Then  by  (2.1)  we  have 


(3.2) 


W4(t) 


-b|(T— t)A» 

(1-e  ) 


-b|(T— t)At 

wt(t)  -  e 


Ve  see  that  wt ,  Wa,  and  W4  all  remain  positive  as  t  decreases. 
We  must  check  to  see  whether  the  Inequality  I4  ^  0  Is  satisfied 
With  the  above  choice  of  w  we  have 


(3*3) 
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(T— t)^a 


to? 


(1- 


— bi  (T— t)/ba 


The  quantity  on  the  right  side  of  this  equation  la  positive  for 
T— t  small  but  Is  negative  when  T— t  Is  large.  Let  t^  be  the  valw 
of  t  for  which  the  right  side  becomes  zero.  Then  T— t^  Is  the 
solution  of  the  equation 


(3.M 


T-t, 


(b4 


— bi  (T— tQ)/ba  ^ 


Thus  we  see  that  at  t^  we  must  abandon  one  of  the  equations 
ii  >  0  and  la  -  0.  Let  us  try  to  choose  w  so  that  It  •  0  and 
14  •  0  before  t^.  We  have 


(t„-t)A4 


(3.5) 


W4(t)  -  W4(to)e 


WaCt)  -  1  -  biW4(tQ)e 


(t--t)A4 


To  verify  that  la  ^  0  we  compute  its  derivative.  We  find 


(3*6)  ■  bg  4-  Wa  •  1  —  (bi  4-  5^)w4(tQ)i 


(t--t)A4 


A  condition  sufficient  to  Insure  that  la  ^  0  is  that  ^  0  for 

all  t  ^  tg.  This  will  hold  if 


(3.7) 
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which  by  (3*2)  and  (3.^)  is  equivalent  to  T— t^  >  b4.  This  last 
Inequality  can  be  checked  by  putting  b4  In  place  of  T— t  In  (3.3) 
and  verifying  that  the  quantity  thus  obtained  Is  positive.  We 
have 

1  r  V.  •4)ib4/ba 

(3.8)  ^(b4  +  )“84  J 


b? 


btb4/^s 

e  -(!•»• 


>  0. 


Hence  fa  ^  0  for  all  t  ^  t^  with  the  above  choice  of  w. 

We  also  see  from  (3*5)  that  W4  and  W3  remain  positive.  Thus 
(3.5)  will  give  a  satisfactory  choice  of  w  until  Ws  becomes  zero. 
Let  tj  be  the  value  of  t  when  this  happens.  Then,  by  (3.2)  and 
(3*^)  we  have 


(3.9) 


(t^— ti)/V)4  b4  +  bs/bi 

'  J  - 


Before  tt  let  us  see  whether  we  can  choose  Wa  «  0  and  have 
14  «  0.  We  see  that  W4  >  0  and  wg  >  0.  We  have  ■  ba  <  0 

so  that  la  >  0,  and  <  0  so  that  It  >  0.  Hence 

this  choice  of  w  will  be  valid  for  all  t  ^  t|. 

Our  basic  solution  Is  suimsarlzed  In  the  following  table. 

This  table  also  Hats  the  properties  thnt  a  c  paired  with  this 
w  solution  must  have.  Any  z  with  these  properties  gives  a  policy 
which.  If  feasible  (l.e.,  satisfies  the  z  constraints),  is  optlsial. 
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t<t^ 

<1>  0 

0 

<3=0 

<3  ■  0 

<2>  ° 

*2*0 

<2>0 

*2=0 

^  »  0 

<3=0 

<3.0 

<3  =  0 

o 

II 

>  0 

*4*0 

II 

o 

W2  >  0 

*1**2 

W2  >  0 

*1**2 

V3;  0 

.3.0 

«3>  0 

Z3=0 

V3  >0 

Z3.O 

*3  *  *4 

«4  >  0 

•3**4 

0 

*3  “‘4 

nc.  1 


Let  us  see  how  this  table  can  be  used  to  obtain  a  partial 
solution  of  the  auto-«teel  problen.  For  the  aovent  let  ua  aaaume 
cs  >  0.  For  t  <  tt  we  must  have 

(3.11)  zi  -  0,  **  -  0,  Za  -  x*.  *4  -  X4/b4. 

For  t|  <  t  <  tjj  we  Bwst  choose 

(3.12)  *1  -  Xt,  Z2  -  0,  *3  -  X4,  *4  - 

This  can  be  done  If  and  only  If  x^itt)  —  b|X«(t|)  ^  0.  Let  us 
assume  that  this  Inequality  Is  satisfied.  Then  for  t^  <  t  <  T 
we  nnist  have 


b» 


(3.15) 


Xi  •  Xa,  Xm 


Zs  ■  >4,  Z4  ■  0 
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whlch  Is  possible  provided  X4(ti)  —  btXc(tt)  ^  0.  Thus  we 

see  that  for  certain  Initial  conditions  we  can  obtain  the  optimal 

solution. 

^4.  The  Modified  w  Solution. 

As  already  has  been  noted,  we  run  Into  trouble  If  X4(ti) 

-  btX2(ti)  <  0.  To  handle  this  ease  we  consider  a  modification 
of  the  basic  w  solution  of  Fig.  1  above.  Let  u^  be  In  the  Inter¬ 
val  such  Uq  we  define  a  solution  as  follows: 

For  Uq  <  t  <  T  we  let  w(t)  be  the  same  as  In  the  basic 
solution.  For  t  <  u^  we  choose  w*(t)  -  0.  For  t  <  u^  but  near 
Uq  we  choose  W4(t)  ■  so  that  li  •  0.  This  choice  will  keep 

14  >  0  for  a  tdille  before  Uq.  We  define  Ui  to  be  the  point 
where  I4  becomes  0  with  this  choice  of  w.  For  t  <  Ui  we  choose 
w  so  that  I4  >  0.  It  is  easily  seen  that  this  choice  makes 
it  >  0,  la  >  0,  ws  >  0  and  W4  >  0  for  all  t  <  U|.  Hence,  In 
this  way  we  obtain  a  w  solution  for  each  u^  In  the  Interval 
Qit'Q*  We  observe  that  for  u^^  •  ti,  U|  •  t|  and  this  solution 
Is  Identical  with  our  basic  solution  of  Pig.  1.  Note  that  ui 
depends  continuously  on  Uq.  Since  for  u^  -  T,  U|  •  T  —  b4,  there 
Is  a  w  solution  for  each  ui  In  the  Interval  Qt ,  T-b^ . 

These  w  solutions  together  with  the  properties  of  the  coi^ 
responding  z  solutions,  are  sumawrlied  In  the  following  table: 
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t  < 

"1  <  ‘ 

STTTT 

<1  >0 

«  0 

«  0 

Ij  »  0 

<3  ■  0 

Zj  =  0 

<2?  0 

*2*  ° 

^  ?  0 

>2*0 

^  *  0 

=  0 

<3.0 

I3  *  0 

<3  »  0 

M 

0 

*4»0 

<4*0 

<'4>0 

*4-0 

^=0 

=  0 

»l“*2 

»J>0 

»l'*2 

«3  >  0 

Xj  *  0 

W3  S  0 

Uj  >  0 

X3  w  0 

W3  P  0 

X3  »  0 

«4>C 

•3  **4 

«4  >0 

*3“  *4 

«4>  0 

•3  “4 

«4  >° 

*3  **4 

Slnc*  !■  «  delta  funetlon  at  we  aust  have  z^(u^  )  s  0, 


n<.  2 


Note  that  If  Uq  >  tg,  then  there  is  no  t  satisfying  the 
conditions  of  the  third  eoluam;  and  If  •  T  then  there  la  no  t 
satisfying  the  conditions  of  the  last  colunn  either. 

35*  The  Equilibrium  Solution 

A  policy  which  seens  plausible  In  sobm  Instances  Is  the 
following:  Make  an  Initial  adjustment  to  bring  down  to  aero 
In  such  a  way  that  after  the  adjustsient  X4  •  btXc.  If  this  Is 
done,  after  the  Initial  adjustawnt  no  Increase  In  eapaeltlea  Is 
necessary  and  all  available  steel  can  be  used  for  auto  production. 
Such  a  policy  would  require  for  the  w  palrd  with  It  that  la(0)  ■  0 
and  i«(0)  •  0,  because  In  general  both  a*  and  S4  will  have  to  be 
delta  functions.  We  shall  construct  a  w  solution  with  this  property. 

First,  we  note  that  our  basic  w  solution  has  this  property 
when  T  Is  such  that  t^  ■  0.  This  suggests  that  we  try  to  chooae 


(5.1) 
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-*)i(T-t)As 

W4(t)  -  oe  +  p 

where  a  and  g  are  constants.  If  ws  Is  chosen  ao  that  fi  ■  0, 
the  Inequalities  (3.1)  becoate 

T 

,  ^  is  2  b2W4(t)  -  (T-t)  b|  /  W4(8)d8  >  0 

(5.2)  ^  t 

i*  5  b4W4(t)  -  /  W4(8)d8  i  0. 

t 

If  tziO)  •  14(0)  •  0,  then 


(5.3) 


**4(0) 


T 


-biT^a 

We  set  E  •  e  and  fro«  (5.1)  —  (5.3)  derive  the  follow¬ 

ing  two  equations  for  a  and  3: 


(5.4) 


bgo  +  (b«  4-  biT)B  •  T 

(ba  +  bib^)  !•  +■  (ba  ♦  b|b4)3  "  T. 


A  solution  of  these  eqxiatlons  will  give  a  w  for  which  la(0)  ■ 
14(0)  *0.  We  have 


(5.5) 


o 


T  1  ba  4  b|T 
^  1  ba  4  b|b4 


-  I  [5i(b4-fl')3 


3 


ba 

(ba  4  b,b«)E 


-  I  Cbad-E)  -  b,b«E3 


1 


•here 
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(5.6) 


Also 


^2  b*  ♦  biT 

(b*  +  btb4)E  ba  ♦  b|b4 


-  (ba  bib4)(ba  -  baE  -  bi 


(5.7) 


/  biT/ba 

■  (ba  ♦  bib4)E  (bae  -  ba  -  biT) 

(ba  bib4)E  (ba  biT  -  ba  -  bxT)  •  0 


Now  let  us  assume  that  T  -  t^  ^  T  ^  b4.  Then  from  (5.5) 

•e  see  that  a  <  o.  Let  us  cheek  to  be  sure  that  for  the  •  we  have 
defined  wa(t),  ^ait) ,  and  W4(t)  are  non-negative  for  0  ^  t  ^  T. 
This  Is  equivalent  to  verifying  that  0  ^  W4(t)  ^  l/bi  and 
“3?^  1  0.  We  have  -JJa  "  “  e-»>i(T-t)ba  ^  q 

be  sufficient  to  cheek  that  W4(T)  ^  o  and  W4(0)  ^  lAi .  Since 
“  ^o  i  conclude  from  (3.^)  and  5.3)  that 


<5-®>  -  T^FTTT  S  -S^S?BT  <  -  lA. 


and 

(5.9)  W4(t)  -  a  ♦  e  -  b,  ]jb4  baAi )(!-«)  -  t]  ^  0 

We  also  must  cheek  that  for  0  ^  t  ^  T,  la  ^  0  and  14  ^  0. 
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Slnce 


(5«10)  •  b2  ^*4  +  1  —  btW4(t)  ■  1  —  bi3 


and  i»(T)  •  b*W4(T)  ^  0,  we  have  la  ^  0  for  all  t  In  Slml 

larly,  we  know  that  l4(T)  -  b4W4(T)  ^  0.  Hence,  If  we  show  that 
-ail*  s  0"  •'  will  have  proved  that  I4  ^  0  for  all  t  In  0^,'0* 

We  have 


(5*11)  -3^  -  a(b4  +  -5^  )  0. 

This  completes  the  proof  that  the  w  which  we  have  defined  Is  a 
solution.  Its  properties,  together  with  those  a  z  pelred  with  It 
must  have,  are  sumaarlzed  In  the  following  table: 


(5.12) 


t  *  0 

0<t<T  1 

^  *  0 

fj  «  0 

<2  >  0 

^  w  0 

t^.o 

m  0 

*4>° 

•4“° 

W2>  0 

■l**2 

«3>0 

*3-0 

w^>0 

*3  **4 

ng.  3 

Istet  ms  oeliitlen  Is  valid 

ealjr  tor  T  ^  b 

1 
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56.  A  Short— tlB»  w  Solution 

The  w  solution  which  we  construct  next  will  be  useful  In 
finding  the  solution  of  our  maxlmun  problem  irtien  the  total  time 
Is  short,  T  <  b4.  This  solution  differs  from  those  already 
constructed  In  that  It  allows  Xs  to  be  positive  and  Xa  to  be  a 
delta  function  concentrated  at  0. 

For  0  ^  t  ^  T  let  W4(t)  «  Wa(t)  •  1  -  bj^'  where 
0  <  ^  <  lAi.  Then  ii(t)  -  0,  l4(t)  >  0  for  0  i  t  <  T.  Also 

(6.1)  ia(t)  «  ba  y-  (T-t)(l-bi^)  -  (^a+b|(T-tn  Y  -  (T-t) 

Now,  If  we  choose  V-  then  la(o)  -  0  and  ia(t)  >  0  for 

t  >  0.  Thus  we  obtain  a  solution  of  the  system  of  Inequalities 

(3.1) .  It  Is  sumirlzed  below  together  with  the  properties  a 
z  paired  with  It  must  have. 


(6.2) 
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Description  of  Solution  and  Proof 

We  now  can  give  the  complete  solution  to  the  original 
problem.  There  are  quite  a  few  cases  that  we  must  consider 
separately.  The  critical  values  t^  and  t,,  which  are  defined 
^>7  (3-^)  end  (3.9),  depend  on  T,  but  in  such  a  way  th^^t  for 
fixed  bi,  ba ,  and  b4,  T  —  t^  and  T  —  ti  are  constants. 

Ca£e  I:  T  is  large  enough  so  that  ti  ^  0.  In  this  case  we 
choose  Z4  to  be  a  delta  function  concentrated  at  0  to  bring  X3 
down  to  zero  Immediately.  This  means  that  if  the  total  time  is 
long  enough  we  should  not  keep  any  steel  in  storage  but  should 
be  using  it  to  build  more  steel  plants.  The  use  of  the  delta 
function  is  pezvlssible  because  I4  •  0  for  t  near  0.  For 
0  <  t  <  t|  we  let 

(7.1)  Zi  •  0,  Za  0,  Sa  -  X4,  84  ■  X4A4 

thus  keeping  Xs  at  zero  level.  At  ti  we  must  distinguish  dif¬ 
ferent  subcases: 

(7.2)  X4(t.).b.x.(t.)i0 

IB:  X4(t,)  -  b,x,(t|)  <  0 

In  case  lA  we  can  produce  autos  at  capacity  without  run¬ 
ning  out  of  steel.  Hence  we  let 


zi  -  xe,  zm  •  0,  zs  •  X4,  84* 

04 


(7.3) 
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for  t|  <  t  ^  tg;  and  for  <  t  ^  T  we  let 

(7.4)  2,  .  xa,  Zs  -  *^~^***  ,  Za  -  X4,  Z4  •  0. 

This  solution  for  Case  lA  Is  optlaial  because  It  can  be  paired 
with  our  basic  w  solution  of  Pig.  1. 

In  Case  IB  we  do  not  have  enough  steel  to  produce  autos  at 
capacity.  Hence  we  continue  to  produce  no  autos  for  t  >  ti,  1. 

(7.5)  Z|  -  0,  Z8  -  0,  Za  -  X4,  Z4  -  ■Ti- 

We  do  this  until  X4  —  bix*  becosws  zero  or  t  •  T-b4,  whichever 
happens  first.  If  X4  —  bix*  becomes  zero  at  t*  then  we  choose 
Zt  "  Xs,  Zg  ■  0,  Za  ■  X4,  Z4  ■  0  thereafter.  This  solution  Is 
seen  to  be  optimal  by  pairing  It  with  the  w  solution  of  Pig.  2 
for  which  Ui  •  t*.  As  we  have  already  remartced  there  Is  such 
a  solution  no  matter  what  t*  Is,  so  long  as  ti  <  t*  ^  T  —  b4. 
If,  on  the  other  hand,  X4(T-b4)  —  b|X8(T-4>4)  <  0  ,  then  for 
T  —  b4  <  t  ^  T  we  choose 

(7.6)  Zt  -  Zs  -  0,  Za  -  Z4,  Z4  -  0. 

This  solution  can  be  seen  to  be  optimal  by  pairing  It  with  the 
w  solution  of  Fig.  2,  for  which  Uq  •  T,  Ut  ■  T  —  b4. 
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C«e«  II;  T  Is  such  that  ti  ^  0  ^  t^.  As  before  we  choose  *4 
to  be  a  delta  function  concentrated  at  0  to  bring  Xa  down  to 
zero  Immediately.  Thereafter  the  solution  Is  as  before.  There 
are  two  subcases: 

IIA;  X4(0)  —  biXsCo)  >  0 

(7.7) 

IIB;  X4(0)  -  b,x»(0)  <  0 

In  Case  IIA  we  let  Zi  •  xg ,  l.e. ,  produce  autos  at  capacity. 
We  use  the  remaining  steel  to  Increase  steel  capacity  before  t^ 
and  to  Increase  auto  capacity  after  t^.  That  la,  for  0  <  t  <  t^ 
we  let 

(7.8)  Zt  -  Xg,  Zg  -0,  Za  a  X4,  Z4  a 


and  for  t  >  t^  we  let 


(7.9)  z,  a  x#,  Zg  a  ,  *3  .  X4 .  Z4  -  0 

This  solution  la  optimal  because  It  can  be  paired  with  our  basic 
solution  of  Pig.  1. 

Case  IIB  Is  similar  to  IB.  The  same  prescription  holds, 
and  the  solution  Is  paired  with  one  from  Pig.  2. 

Case  III:  T  Is  such  that  5^  0  ^  T  —  b4.  There  are  three 


subcases 
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IIIA:  C4  —  bjCa  ^  bi  -H- 

(7.10)  IlIB:  C4  -  b,ca  <  -  -|J 

IIIC:  i  C4  -  b,ea  < 

In  Case  IIIA  we  use  our  Initial  stockpile  of  steel  to 
Increase  auto  capacity,  l.e.,  we  let  Zg  be  a  delta  function  con¬ 
centrated  at  0  bringing  Xg  down  to  zero.  Thereafter,  we  let 
Zi  «  Xs  and  use  any  reoMlnlng  steel  to  Increase  auto  capacity, 
l.e., 

(7.11)  zi  -  xt,  **  -  .  Za  -  X4,  X4  -  0 

This  solution  Is  optlaal  because  It  can  be  paired  with  the  basic 
w  solution  of  Pig.  1. 

In  Case  IIIB  we  find  ourselves  short  on  steel  capacity.  The 
policy  and  proof  are  the  sane  as  In  Case  IB. 

In  Case  IIIC  we  can  oiake  an  Initial  adjustment  so  th'^t  Xg 
becones  zero  and  X4  •  b|Xa«  We  do  this  by  choosing  Za  and  Z4 
to  be  delta  ftinctlons  concentrated  at  0.  After  that  we  let 
Zi  •  Xa,  Za  •  0,  Za  ■  X4,  s«  ■  0.  This  solution  Is  optimal 
because  It  can  be  paired  with  the  equlllbrliia  w  solution  of 
Fig.  3. 

Case  IV:  T  ^  b4.  There  are  three  subcases  which  depend  on  the 
Initial  values: 
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IVA;  C4  -  b,C2  ^ 

(7.18)  IVB:  C4  -  b,ca  > 

IVC:  i  C4  —  b|C*  <  C3 

In  oase  IVA  the  solution  and  proof  are  the  sane  as  In  Case 
IIIA. 

In  Case  IVB  we  choose  z»  -  0  and  Z4  -  0  for  all  t.  As 
always  we  let  Za  •  X4.  We  choose  Zi  In  any  way  such  that 
*i(t)  i  x»(t)  and  X3(t)  ■  0.  Thus,  In  this  case  the  solution 
Is  not  unique.  Any  solution  of  this  fom  can  be  seen  to  be 
optimal  by  pairing  It  with  the  w  solution  of  Pig.  8  for  idilch 

«o  "  T* 

In  ease  IVC  we  find  ourselves  In  an  Intensedlate  ease,  unable 
to  follow  the  policies  suggested  by  IVA  and  B.  In  this  ease 
we  make  an  Initial  adjustment  of  the  steel  stockpile  down  to 
the  value  ci,  using  this  steel  to  increase  auto  capacity.  There¬ 
after  we  choose  Zi  ■  xa«  z*  ■  0,  Z3  •  X4,  and  Z4  •  0.  The 
value  e.  Is  determined  so  that  X3(T)  -  0.  It  is  found  that 

(7.13)  ci  - 

has  this  property.  This  solution  Is  optimal  because  it  can  be 
paired  with  the  short— time  w  solution  of  Pig.  t. 


I 
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Inltial  Adjustaants 


DBQ3 

nil  to  <  0  £  T  - 

nnasm 

■ 

Adjust  x^  to  0  by 

inereasiog  x.. 

4 

Being  x.  to  0  by 
increasing  x^ 

*Build  snto  Mpaeity*. 

B 

* Build  steel  capacity*. 

C 

No  Case 

No  Case 

Adjust  so  a-  ■  0, 

*4  •  ^ 

increasing  and  x^. 

Adjust 

downward, 
but  net  to 

0,  so  that 
«5(T)*  0, 
IterMse 

After  the  Initial  adjustaente  the  optiml  policy  can  be 
determined  by  a  priority  ayatem.  Before  t,,  building  ateel 
capacity,  i.e.,  Z4,  has  firat  priority.  Thia  continues  after 
ti  until  either  X4  ^  btXs  or  t  -  b4,  whichever  comes  first.  When 
this  happens,  which  may  be  at  tj,  of  course,  firat  priority  is 
given  to  auto  production,  Z|.  This  will  use  up  all  available 
unless  X4(t|)  >  btXs(tt).  In  that  ease  second  priority 
is  given  to  building  steel  capacity  until  the  time  t  .  After 
tjj  second  priority  Is  given  to  building  auto  capacity. 
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STUDY  II: 

A  MATHEMATICAL  MODEL  OF  TWO 
INTERDEPENDENT  PROCESSES 

by 

Richard  Bellnan 
R .  Shenaan  Lehman 


51.  Introduction 

In  this  paper  we  shall  study  a  simple  mathematical  model 
of  the  Interaction  of  two  processes. 

Using  the  fundamental  device  of  the  theory  of  dynamic  program¬ 
ming,  we  shall  convert  the  original  problem  Involving  the  maxi¬ 
mization  of  an  integral  subject  to  integral  constraints  into 
that  of  solving  a  functional  equation  which  may  be  converted 
into  a  nonlinear  partial  differential  equation. 

Although  the  over-all  approach  is  straightforward, the  details 
of  the  analysis  are  not  simple  and,  in  general,  a  very  precise 
argumentation  is  required. 
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§2.  Formulation  of  the  Problem. 

We  shall  consider  the  following  maxinilntion  problem;  Given 


dxa 

—  -  aaZa  —  Z3, 

Xa(0)  -  Ca 

“  baJaZa  ~tgZa , 

*3(0)  -  C3 

where  Z2  and  Zs  are  functions  of  t  subject  to  the  constraints 

za(t)  +  Z3(t)  ^  Xa(t),  z,(t)  ^  0 

(2-2)  ^  . 

Oa**(t)  -f  j^ZsCt)  ^  Xs(t),  *3(1)  ^0, 

deteimlne  Zg  and  Z3  so  as  to  rnaxlmlze  Xa(T).  The  constants  aa,  b3, 
^8,^3  are  all  positive  as  are  the  Initial  constants  Ca  and  C3. 

This  problem  Is  equivalent  to  the  functional  equation 

(o  ^<**<»>*  f(ca,cs,0)  -  c,. 

(2.3)  D[0,g 

where  Xa  and  X3  satisfy  (l)  and  the  maxlaum  Is  over  all  choice  of 
the  decision  variables  Za  and  z,  which  satisfy  (2)  on  the  Interval 
[C.s^,  and  Max  x»(t)  -  f(ca,c3,T),  cf.  [1]. 

A  solution  of  the  partial  differential  equation 
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(4) 


where  D[q]  represents  a  choice  over  all  z^(0)  satisfying  (2),  will 
yield  a  solution  of  the  functional  equation  and  hence  the  original 
maximum  problem. 

Our  procedure  Is  to  assume  the  existence  of  a  solution  of  (4), 
a  function  f(c2,C3,t)  with  continuous  partial  derivatives  In  Its 
three  arguments  and  piecewise  continuous  second  derivatives.  Ha 
assume  also  that  the  Sj^(t)  for  this  maximum  function  are  piecewise 
continuous.  Using  these  assumptions  we  derive  the  fom  of  the 
solution. 


It  Is  convenient  to  look  at  the  question  In  tenns  of  regions 
in  the  (r,T)  plane  where  T  Is  the  time  remaining  before  the  process 
ends  and  r  Is  the  ratio  Xa/xa  or  Ca/ca.  A  solution  to  (4)  and  also 
(3)  Is  uniquely  determined  by  the  specification  of  what  policy  Is 
to  be  followed  for  each  couple  (r,T).  For  given  T  the  optimal 
policy  depends  only  on  the  ratio  because  of  the  homogeneity  of  (l) 
and  (2). 

In  the  quadrant  r  ^  0,  T  ^  0  ms  can  find  certain  regions  In 
which  a  given  policy,  for  example,  Zt  -  Xa,  Zs  «  0,  Is  to  be 
followed  at  t  •  0.  The  problem  reduces  to  locating  the  boun— 
darles  of  these  regions. 

S3.  B/A  Is  a  Monotone  Function  of  r. 

First,  let  us  look  at  the  partial  differential  equation  (4). 
The  right  side  is  a  linear  form  In  the  variables  Za,  Za.  Let 


The  location  of  the  maximizing  point  (za.z,)  will  depend  on  the 
ratio  B/A.  Our  first  aim  Is  to  consider  this  ratio  B/A  as  a  func 

tlonofr  andT.  Let  f(c..c,.T)  -  f.(c./c,.  c,.  T).  -men 


>f 


if 

»Ca 


-  1  jfi 

C3 


■  n 


Vj 

dr 


^  . 

>53 


Also  because  of  the  homogeneity,  ft(ca/c3,  aca,  T)  -  af,(c,/c3,  c,. 
and  hence  a 3  •  Consequently 


Clearly  B/A  Is  a  function  of  r  and  T  alone.  For  simplicity  we  con. 
slder  A  and  B  as  functions  of  r  and  T  by  setting  c,  -  1,  and  we 
drop  the  subscript  on  f.  We  have 


*  •  - &(f  -  r 

B  -  b,  J,  (f  -  r  ^)  -  |f. 

considered  for  fixed  T,  A  Is  a  monotone-decreasing  function 
of  r.and  B  Is  a  monotone-lnereaalng  function  of  r.  To  prove  this 
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we  differentiate  with  respect  to  r  and  find 


•  (sa  +  2far) 

ar* 


dA 

-S¥- 


■4?*  •*(!  +  ^3^3**) 


It  Will  be  sufficient  to  prove  <  o  because  the  flrat  derl- 

vatlves  of  f  are  continuous.  This  will  follow  from  the  fact  that 


,(r)  i  f(r^A)  f  f(.r^  _ 

2 

To  prove  this  we  note  that  a  possible  policy  for  time  T  with 
Initial  values  re®,  C3  la  the  mean  between  the  policies  which  are 
optimal  with  Initial  values  (r4-A)c3,  Cs  and  (r-A)c3,  C3  for  the 
same  time  T.  This  Is  true  because  both  the  equations  and  restric¬ 
tions  are  linear. 

§4.  The  case  Ss/ia  <  1 

Let  us  begin  by  studying  the  case  In  which  Tfs/H  <  !•  If  ws 
consider  the  constraints  (2)  In  the  Zg,  23  plane,  we  find  there 
are  three  cases  depending  on  the  position  of  the  lines 

I*:  Za  *3  “  *2 

is!  ifa*»  ♦  ^9*3  "  *3- 

The  line  la  may  be  above  I3  «o  that  la  Is  the  only  essential  con¬ 
straint.  The  lines  may  cross  so  that  both  are  essential.  The  line 
la  may  be  below  is  so  that  It  Is  the  only  essential  constraint.  In 
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each  of  these  three  cases  we  denote  the  vertices  of  the  convex 
set  defined  by  (2)  by  P,  P‘ ,  q,  r,  r* . 


For  P';  -  XsA»,  Ss  -  0 

R':  Za  -  0,  Zs  •  Xs/^a 

Q!  *2  ■  Za  •  *3— ^X»/Ja-*t 

P:  Za  -  xa»  Za  -  C 

R:  Za  -  0,  Zs  >  Xa 

The  linear  form  In  the  right  side  of  (4)  will  In  general  be 
maximized  by  choosing  (za,Za)  to  be  a  vertex.  However,  If  the 
ratio  B/l  -  1  or  83/4  the  maximizing  point  (z,,2a)  Is  not  uniquely 
determined  by  the  linear  form.  Consider  the  trajectory  curves 
for  a  given  solution  In  the  (r,T)  space,  that  Is,  the  curves  deter¬ 
mined  by  ,  T-s)  as  8  runs  from  0  to  T.  If  the  trajectory 

never  runs  along  one  of  the  curves  B/A  -  1  or  B/A  -  8a/4,  then  at 
each  point  an  optimal  policy  will  be  to  choose  (za,Za)  to  coincide 
with  one  of  the  vertices.  A  policy  of  choice  of  an  intermediate 
point  can  only  be  optimal  on  a  curve  In  the  (r,T)  plane  and  not  in 
a  region. 
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Except  for  this  disturbing  possibility  of  choice  of  an 
Intermediate  point.  If  we  knew  the  location  of  the  curves  B/A  ■  1 
and  3/A  -  we  would  know  the  optimal  policies.  They  are 

as  follows: 

r  >  1/^3  1/ia  <  r  <  1/6 

R'  B  >  A;  R 

A:  P*  A>B>  4jA:  , 

B<-|^A:  P' 

It  la  clear  from  the  differential  eqiuatlon  (1)  that  for  T 
sufficiently  small  the  optimal  policy  Is  to  maximise  s*  and  take 
Zs  ■  0.  The  favorable  effect  of  taking  Zs  positive  la  of  second 
order  while  the  unfavorable  effect  of  decreasing  la  of  first 
order.  This  means  that  the  optimal  policy  Is  P  when  r  <  l/2ra  And 
P'  »rt»en  r  >  l/iTf 

$5.  Region  Where  P*  Is  Followed. 

Knowing  the  solution  for  small  T,  we  use  the  Information  thus 
obtained  to  extend  the  solution  to  larger  T.  First  we  treat  the 
case  In  which  r  >  l/gg  and  P*  Is  followed.  There 

-§f» - 

SO  that  XsCt)  •  Cae  x»(t)  ■  c»  +  CaCl-e""^).  Consequently 

f  (®#»C3»T)  ■  Cs  +  — ^  Cs(l— e  '^) 


r  <  1/ia 
B  >  A:  R 
B  <  A:  P 


I 
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and 


— T 

A  B  Eae 

E  -  d-e-'^)  -  1. 


We  wish  to  find  iriien  B 


Then  we  have 


ba^a,-4 

b303a2+’  Bsas 


The  necessary  and  sufficient  condition  that  there  be  such  a  T  Is 
that ^  S  bsVsar  —  I*  >  0.  From  (l)  one  sees  that  following  P'  does 
not  decrease  r.  Define  T*  by  the  equation 

--T*  _  'X 


If  the  right  side  is  positive,  and  let  T*  ■  oo  otherwise.  Then  for 
r  >  l/lat  T  <  T*,  we  know  the  solution  completely.  We  note  for 
future  reference  that  this  result  holds  also  when  Tfa/gg  ^  1. 

Let  us  look  at  the  region  l/ta  <  r  <  1/84.  Because  of  the 
continuity  of  B/A  we  know  the  policy  Q  Is  followed  for  values  of  T 
that  are  slightly  larger  than  T*.  We  now  prove  that  actually  the 
policy  Q  Is  followed  In  the  whole  region  l/Jf«  <  r  <  l/<?3,  T  >  T*. 


§6.  Q  Policy  not  Preceded  by  P*. 

Let  us  first  establish  that  In  an  optimal  policy  It  does  not 

happen  that  first  P*  Is  followed  and  then  Q.  For,  consider  any 

policy  which  consists  In  following  P*  for  0  <  t  <  u  —  6  and  Q  for 

6/u 

u-6<t^u.  We  prove  that  for  6  and/  sufficiently  small  there 
Is  another  policy  which  Is  superior  having  the  form  Q  for  0  <  t  <  A 
and  P*  for  A  <  t  <  u,  where  A  Is  a  positive  numberl 
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Let 


j  /  ^3  + 


— (aa+1 ) 

b3Jj”3+Y» 


Using  vector  matrix  notation  with  x(t) 


c 


» 


we  have  -  P»x(t)  when  P»  Is  followe(i  and  g^"  “ 

Q  Is  followed.  Thus  for  the  policies  described  above  we  have 


I:  x(u)  -  -  (I  +  Q6  +  o(6))(l  -  P'b  +  o(6)  )e**''^*c 

-  (I  +  (Q  -  P' )6  +  o(6))e^'“.  c 
II:  x(u)  -  eP'''(I  +  (Q  -  P’)A  +o(4.)).c 


For  6/u,  ^/u,  u  - >  0,  the  differences  between  the  results 

P  ®  U  w 

at  time  u  obtained  by  following  these  policies  and  e  .  c  which 
is  obtained  by  following  P'  for  the  entire  Interval  are 


I:  6  [^(Q  -  P')  (Q  -  P»)P'u  +  o(u)  J*c 

II:  a[^(Q  -  *”)  P'(Q-P')u  +  o(u)  J  ‘C. 

Setting  ail  ■  aay3  aai  ■  — 3ia  is(b3+l) »  **  find 
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(Q  _  pt)  .  _JL 


-(aa  fe+ *  )/!r2\ 

U,-4)  !r,(b,*i)  / 


(5^s-4) 


^ati  -*11/^2 


*2  1  —  aai/ja 


and  also 


(Q  -  P')P'  -  (aa-l)  _ 
( ^3—  4  )^2 


0  at 


0  aa  I 


/*  /**  -aa/^a  \ 

P»(Q  _  p*)  .  / 

(53-4)  j  y 

The  difference  In  lower  components  at  time  u  Is  thus 


4y~  ~  Cs/Ja)  -f  ^  aaiCsU  -f  o(u)^ 


II: 


^  r 

: — |^aai(ca  -  Ca/Ja)  -  aai  (ca  -  C3/ya)u+o(u)  J 


The  difference  In  upper  components  Is 


I: 


II: 


^  j  |^aii(ca  -  Ca/fla)  +  anCaU  +  o(u)J. 

|ati(c2  -  Cs/lfg)  +  — (ca  -  C3/3i)u  +  o(u)  J 
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We  choose  ^  so  that  the  policy  of  type  II  gives  the  same 
value  for  X3(u)  as  that  given  by  the  policy  of  type  I.  Thus 

^  (Cg— €3/^(2)  +  0(^)Z1 

{cs—Cs/lfa)  (l-u) 

Now  we  prove  that  the  quantity  X2(u)  for  II  exceeds  X2(u)  for  I 
by  a  positive  amount.  Since  <  0,  (02-03/^2)  >  0,  and 

(l-u)  >  0,  this  Is  equivalent  to  the  statement 

ail  |jc2-C3/y«)  +  J 

>  ^  (02-^3/12)  +  C3U  +  (a»i  ♦  ^  +  o{u)). 

The  quantity  In  brackets  la  positive  because  u  Is  small.  Hence 
this  statement  Is  equivalent  to 


aii(l— u)  >  (sii  +  1  ^ 


But  a2i  ^ —  +  ail  *  — az  2^3  { h  S'*"  1 )  +  *2^3  +  ^2  ■  —  (a2b3i3— 4)  -^  <  0, 
so  that  the  above  Inequality  holds  for  u  and  6/u  sufficiently  small. 
We  see  that  with  the  above  choice  of ^  the  policy  of  type  II  Is 
superior. 


^7.  a  Policy  not  Preceded  by  F. 

Now,  we  prove  that  an  optimal  policy  cannot  have  the  form; 
Follow  R  for  0  <  t  <  6;  follow  Q  for  6  <  t  <  T  -  T*;  follow  P* 
for  T  —  T*  <  t  <  T.  This  will  complete  the  proof  that  Q  is  followed 
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In  the  whole  region  l/fz  <  r  <  I//3,  T  >  T*.  We  show  that  a 
policy  of  the  above  form  Is  not  optimal  by  comparing  It  with  a 
policy  of  the  form:  Follow  Q  for  0  <  t  <  T  -  T*;  follow  P'  for 
T  -  T*  <  t  <  T. 

As  before  we  let  P*  and  Q  denote  the  matrices  for  which 
dx/dt  “  P*x  and  dx/dt  ■  Qx  when  and  Q  respectively  are  followed. 
Similarly  we  let 

'-1 


) 


^3^3 

For  the  policies  described  above 

I:  X(T)  -  e^(T  '  T*)  ^ 

-  e^'^*  (I  +  (R-Q)6  +  0(6))  •  c 


II:  x(T)  -  e^’-^ 


c . 


NOW 

j  -fe(a8+l)  (sz  +  1) 

”  n&-^T  I  .  .  ^ 

P'T^  0(T— T*)  /Oil  Oia\ 

e  e  ■  Voa  1  '^22)'  ^*^*0  for  the  two  policies  we 

have  respectively 

I:  X2(T)  «  OiiCa  +  ai2C3  +  ^  |oi  i  (*a+l  )-Oi2  ( 03  2(3+ )  J  ♦  o( 


II:  3ta(T)  *  0|]C2  +  Ots^s* 


•34“ 


sine.  (Ke-fc)  <  0.  (  =  ,-«iee)  >  0,  It  will  6.  sufflelent  te  pro,* 


that 


~  Oia  )  >  0. 


Note  that  an  and  ata.  although  they  depend  on  T.  do  not 
depend  on  r.  Consequently.  If  we  succeed  In  verifying  the  Inequality 
fer  a  particular  r  for  which  l/jTa  <  r  <  l/lfa.  we  will  have  proved 
the  desired  result.  There  Is  a  value  r*  In  this  Interval  with  the 
property  that  If  Q  Is  followed  and  ca/ca  -  r*  then  Xa(t)/X3(t)  -  r* 
as  long  as  Q  Is  followed.  We  postpone  the  proof  of  the  existence 
of  such  an  r*  for  the  moment  and  continue  the  above  proof,  with 
the  additional  assumption  that  ca/ca  -  r*.  In  this  particular 


case 


^Q(T-T*)  _ 


(T) 


0  yiA.(T) 


where yM(T)  >  0  depends  only  on  T.  Since 

Xa(T)  «  Xa(T-T-)  +  X3(T-T*)  ^ 


Xa 


we  have 


where  the  asterl8.<s  represent  numbers 


In  which  we  are  not  Inter- 


I 


ested . 
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Consequently 


.P'T'  0(T-T*) 


We  want  to  prove 


(a>8^  *  ^a) 
y2jr3(  b3+l ) 


yU(T)  r  >  0- 

^  r2^3(b3-H)-l 


or 


^2  ^  (ag+l )  ( b3+l )  —  (b3jf3-f  iTz  )  (sa  ^3-f  ^  ) 


>  0 


or 


(  ^2— ^3  )  (a2b3  ^3— Ib  )  «  (^2  — 83)^  >  0 

which  holds  since  >  ^3  and  Pi  >  0.  Except  for  the  proof  of 
the  existence  of  r*  this  completes  the  proof. 

§6.  Effect  or.  r  of  Q,  Polity. 

Let  us  Investigate  the  effect  or.  the  ratio  r  »  Xa/xs  of  follow¬ 
ing  Since  this  will  be  Important  also  In  the  case  If 3/^3  >1, 
we  consider  both  cases  at  the  same  time.  Since  xa  ■  ns*  w«  have 

-S*- - -al- *  ^  • 


Now,  If  we  let 
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—(*2+1 ) 


v  baira-J-jra/ 


ai  i 

ata 

a2i 

aaa 

we  have 


*» I *2  +  ® 12*3 


dr 

“STF 


*a(J  3— & ) 


+  rasiXs  +  ra22X3 


or 


-(«3-r2) 


dr 

“arr 


■  azir*  +  (*22-«ii)r 


ai2 . 


The  zeros  of  the  rlgnt  side  are 


»l l-®22 

Saz  t 


+ 


®21 


There  Is  one  positive  zero  and  one  negative  zero.  We  define  r* 
to  be  the  positive  zero 


r* 


-*22 

®21 


+ 


^)»  4  -ili 

1  *2 1 


(6a-®2 ) 
2l2lbs^l) 


*4- 


frr) 


TJTT 


Also  r*  Is  between  l/Ya  and  1/^2.  The  easiest  way  to  verify  this 
la  to  compute  the  value  of  the  quadratic  on  the  above  right  side 
for  r  •  l/fe  and  r  ■  l/^.  The  values  are 

-  ^  (Dj.1 )  (fa-y, ) 

t.  r. 


which  differ  In  sign. 

When  the  Q  policy  Is  followed  and  r  •  r  does  not  change. 

It  la  also  easy  to  see  from  the  differential  equation  above  that 
for  <  1,  r  Increases  when  r  <  r*  and  r  decreases  tfhen 

r  >  r*.  On  the  other  hand,  for  >1*  r  Increases  when 

r  >  r*  and  decreases  when  r  <  r*. 

§9-  Remaining  Regions. 

We  finally  conclude  that  Q  Is  followed  in  the  entire  region 
l/^a  <  r  <  l/Ks,  I  >  T*.  It  follows  that  In  this  aaine  region 

<  B/A  <  1.  Since  B/A  Increases  monotonely  as  r  Increases, 
we  can  conclude  that  B/A  <  1  for  r  <  l/ga,  T  >  T»  and  B/A  > 
for  r  >  l/jJa,  T  >  T*.  From  this  we  conclude  P  Is  followed  In  the 
region  r  <  T  >  T*  and  R'  In  the  region  r  >  l/^a,  T  >  T*. 

The  solution  has  now  been  found  everywhere  except  In  the 
region  r  <  l/^a,  T  ^  T*.  A  natural  conjecture  Is  that  P  Is  followed 
In  this  region  also.  This  can  be  checked  directly,  but  It  also 
follows  from  the  monotonlclty  of  B/A.  We  know  that  for  T  <  T*, 

>  l/?a»  the  function  B/A  <  ifs/ia  <1.  Hence  In  the  region 
T  <  T*,  r  <  1/1^2,  B/A  <  1  and  P  Is  followed. 


'3ft- 


T* 


P* 


r 


§10.  The  Case  iTs/iTz  >  !• 

Now  we  consider  the  case  In  which  ^3/8*  >  !•  Again  there 
are  three  different  cases  depending  on  the  position  of  the  lines 


fs :  Zs  +•  Z3  “  Xa 

-^3*  J**a  ^^3^3  “  X3. 


We  use  the  same  letters  to  denote  the  vertices  as  in  the  ease 
^s/i^a  <  1* 


Za 
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The  optimal  policies  are  determined  by  the  ratio  as  follows: 


r  >  l/U 

l/jfs  <  r  <  1/^, 

r  <  l/ifa 

B  >  A:  R- 

5  >  ^  A:  R« 

B  >  A:  R 

V 

B  <  42-  A:  P» 

A  <  B  <  A:  Q 

3  <  A:  P. 

3  <  A:  P 

The  results  obtained  In  the  previous  case  shows  that  for 
r  >  l/Jfz,  T  <  T*,  the  optimal  policy  Is  to  follow  P».  There  Is 
another  Important  point  which  can  be  calculated  using  those  results, 
the  point  on  the  line  r  -  \/'iz  where  B  »  A.  Let  Tg  be  that  point 
If  It  exists  and  otherwise  let  T^  ■  oo .  We  have 

a2e“^°  =  _  ^"'o^  _  ^ 

e  °  «  aat>3  4  .  ^ _ 

^2  ( b 3 J’a  ) 

A  necessary  and  sufficient  condition  that  T^  f  oo  Is  0.  Also, 
when  ^  >  0,  T^  <  T*  because  jfs  > 

Bacause  of  the  monotonlclty  of  B/A  as  a  function  of  r,  B  <  A 
In  the  region  T  <  T^,  r  <  1/5* ;  and  hence  the  policy  P  Is  followed 
there.  Our  next  aim  la  to  determine  the  location  of  the  region 
where  Q  Is  followed. 

§11.  Q  Policy  Does  Not  Precede  P. 


First,  we  establish  that  In  an  optimal  policy  It  does  not 
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happen  that  first  Q  Is  followed  and  then  P.  The  method  Is  the 
same  as  that  used  previously.  Consider  any  policy  which  consists 
In  following  Q  for  0  <  t  <  6  and  P  for  6  <  t  <  u.  We  prove  that 
for  6  and  5/u  sufficiently  small  there  Is  a  superior  policy  having 
the  form  P  for  0  <  t  <  u  —  A  ®nd  Q  for  u  —  A  <  t  <  u  where  A  is 
a  positive  number. 

Let 

,.f“  "V 

oj 

and  Q  be  the  matrix  for  which  dx/dt  »  Qx  when  Q  Is  followed.  For 
the  policies  described  above  we  have 

I:  x(u)  ■  e^^  •  c  -  +  (Q— P)6  +  o(6))  *  c 

II:  x(u)  =  •  c  -  (I  +  (Q-P)A  +  o(6))e**'^  •  c 


For  6/u,  A/u,  u  — >  0,  the  difference  between  the  result  at 

P\i 

time  u  obtained  by  following  one  of  these  policies  and  e  *c  Is 


I:  5  j^(Q-P)  +  P(Q-P)u  -*■  o(u)  J  •  c 
II:  A  r  (Q-P)  ♦  (Q-P)Pu  o(u)  "I  •  c 


Setting  ai2  “  — (•z+l)# 


(<i-P)  - 


(3s-l4)  y_i,a*2 


we  find 


I 


-u- 


P(Q-P) 


— ](aaa 

(If 3—1^)  ^  ^1 


Lil_ 


ai2 


*82 


*8 

-fey 

0 

0 


I; 


II: 


The  difference  In  lower  components  at  time  u  Is  therefore 


*88  (c*— Ca iTa  )  —  *l^^a  (ca— Ca2^»  )u  +  o(u)  J 


( ^3— Ja  ) 


( iTa— ) 


*88 (ca— Ca/a )  —  (aa+1 )/8*8aCaU  +  o(u) ^ 


The  difference  In  upper  components  Is 


1: 


II: 


— ^i8{c3-ca^)  aiaaa (ca— CaiTa  )u  +  o(u)] 

^laCcs—caya)  —  *i8(**+l)^8  Ca^  -f  o(u)  J  . 


{63-it) 


We  choose  ^  so  that  the  policy  II  gives  the  same  value  for 
X3(u)  as  that  given  by  policy  I.  We  have 

^  ^  6(C3-Cg^yj  [Agg-*tgygu  o(u)l — 

*28  Oca-ca^a)  —  (*8^l)yaC8u] 

Now  we  prove  the  quantity  Xa(u)  for  11  exceeds  that  for  I  by  * 
positive  amount.  Since  >  0,  (ca-CaJfa)  >  0»  ***  >  thl* 

Is  equivalent  to  the  statement 
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«ia  j^Ccs-c^Ua)  -  («a+l)24cau  +  o(u)  J  |a**-*,*jJaU  o(u^ 

>  a»a  1^1  +  aau  +  o(u)  ja**  jjca-c.ya )  -  (ag+l  )2raCaU  J  ^ 


This  Is  equivalent  to 

jaaa-*iajjau  +  o(u)  J<  aaa  1  +  aau  +  o(u)  J. 

But  aaaaa  +  a^aifa  •  aa  (b32r9-f ila )  —  2fa(aa+l)  •  ^  >  0.  Consequently, 
the  above  Inequality  holds  for  u  and  6/u  sufficiently  small. 

§12.  Q  Policy  Does  Not  Precede  R'. 

Next,  we  establish  tnat  In  an  optimal  policy  It  does  not  happen 
that  first  a  q  policy  Is  followed  and  then  R».  Consider  any 
policy  which  consists  In  following  Q  for  0  <  t  <  6  and  R'  for  6  <  t 
<  u.  We  prove  that  for  u  and  6/u  sufficiently  small  there  is  a 
superior  policy  having  the  form  R*  for  0  <  t  <  u  and  Q  for 

“•A<  t  <  u  ,  where  A  Is  a  positive  number.  Let  R’  ■ 

For  the  policies  described  above  we  have 

I:  x(u)  -  .  c  -  *  (Q-R')6  ♦  o(6))  •  c 

II;  x(u)  -  *  c  -  (l  +  (Q-R')^  +  o{A))e^''^  •  c 


For  6/u,  A/u,  u  - >  0  the  difference  between  the  result  at  time  u 

obtained  by  following  one  of  these  policies  and  e^*'^  •  c  is 

I;  e  |r(Q-R»)  R'(Q-R')u  +  o(u)*J*  o 

II:  A^(a-R*)  ♦  (Q-R»)Riu  ♦  o(u)J*  c. 

Setting  ait  -  ♦  ifa.  ftai  -  HTajsCbrH),  we  find 
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(Q_RI)  ■ 


(^3—^4  ) 


1 


&2  1 


1/^3 

lai/i^s 


R'((^.')  -  ■ 

(1(3-1^) 


-1/6  i/il 

3  -b3/6 


(Q-R'  )R'  -  V- 


^s(y»— 6 ) 


0  an 


0  aa  1 


The  difference  in  lower  components  at  time  u  la  therefore 

;~7  "y-.  I  »8t(Ca-C3//3)  +  a2|b3(c«-C3/l53)u  +  o(u)  L 

(  #3— te  }  •-  -* 

II:  —  r  *2  1  (ca-c 3/1^3)  —  aaj  C3U  -f  o(u)*l  . 

(  O3— #2  )  ^  0  3 

The  difference  In  upper  components  Is 

1:  --T -  r  *11(02—03/^3) - —  (C2-<!3/)r3  )'J  "| 

(03— «b)  W3  “* 


II; 


naiiCca-Cs/jf'a)  -  an  cjU  o(u)  1 

(  ®3— «  )  ^  *3  "* 


We  choose  ^  so  that  the  policy  II  gives  the  same  value  for 
X3(u)  as  that  given  by  policy  I.  We  have 

^  .  6(ca-C3/ir3)  C  H.b3iH-o(u3 

(cg-Cs/j^s) - C3U 

I  3 


We  prove  that  with  this  choice  of  ^  the  quantity  Xa(u)  for  II 
exceeds  Xa(u)  for  I  by  a  positive  amount.  Since  (fe— 9«)  >  0, 
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(cs-cs/ljs)  >  0,  this  is  equivalent  to  the  statement 

Sit  ^  ^  o(u)  ^  P  (cs-ca/lfe)  “  ®3'^  ^ 

>  [a,,  +  o(u)][(c,-C3/)r5)  -  Cau] 


which  is  equivalent  to 


aii(l  4-  bgu  +  o(u))  >  a,i - u. 

Oa 


But 

■  SshaJ^s  +  bsHz  —  iTa  (ba-f  1 )  ■  ^  >  0. 

Consequently  the  above  inequality  holds  for  u  and  6/u  sufficiently 
small. 

§13.  Location  of  Q  Region. 

Moreover,  since  a  Q  policy  Is  net  followed  for  T  <  T^,  we 
conclude  that  a  Q  policy  Is  always  succeeded  by  the  P*  policy. 

Now  we  prove  that  the  region  where  the  Q  policy  is  followed  does 
not  contain  any  points  (r’,T')  where  r'  <  r*.  Suppose  the  con¬ 
trary.  Then,  since,  as  In^,  r  decreases  when  the  Q  policy  is 
followed  for  r  <  r*,  there  are  points  (r,T)  in  the  Q  region  for 
which  r  <  r*,  T  <  Tq.  But  this  contradicts  a  result  which  states 
that  all  such  points  are  in  the  region  where  P  is  followed. 

Consider  the  Piceati  differential  equation  in  §8  which  repre¬ 
sents  the  dependence  of  r  on  t  when  the  Q  policy  is  followed. 
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For  r*  <  <  l/ta  let  ^(r^)  be  the  time  required  to  reach 

r  «  l/iJe  beginning  from  r^.  From  the  differential  equation  we 
see  that  ^(r)  Is  a  monotone  decreasing  function  of  r  and  ^(r)  — >■ 

+  oo  as  r  — >  r*.  Also  ^(l/|$a)  ■  0. 

The  result  that  a  Q  policy  is  always  succeeded  by  a  P'  policy 
permits  us  to  reach  some  conclusions  about  the  location  of  the  Q 
region.  In  fact.  If  (r,T)  Is  a  point  In  the  Q  region  then 

Tq  i  T  -  (t)(r)  <  T*, 

and  consequently 

To  Mr)  ^  T  ^  T*  ^(r). 

There  are  points  (r,T)  In  the  Q  region  with  arbitrarily 
large  T.  To  verify  this  we  consider  the  continuous  curve  on  which 
B/A  -  1  which  passes  through  the  point  (l/^,To)»  Near  the  point 
(1/I2»Tq)  It  forms  one  of  the  boundaries  of  the  Q  region,  and  It 
will  continue  to  do  so  as  T  Increases  since  by  the  above  results 
It  can  never  leave  the  region  r*  <  r  ^  l/J(a* 

§14.  Remaining  Regions. 

In  the  Q  region  1  ^  B/A  ^  By  the  monotonlclty  of  B/A 

as  a  function  of  r,  we  Infer  that  to  the  right  of  the  Q  region 
for  T  >  T*  the  optimal  policy  is  R',  while  to  the  left  of  the  Q 
region  the  optimal  policy  Is  P. 
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This  completes  the  finding  of  the  solution  except  for  one 

possibility.  Vfhst  policy  Is  followed  on  the  boundaries  of  the 

regions?  It  will  make  no  difference  If  a  solution  trajectory 

crosses  the  boundaries,  but  If  the  solution  trajectory  runs  along 

one  of  the  boundary  curves  we  cannot  even  be  sure  that  an  optimal 

policy  Is  obtained  by  choosing  one  of  the  vertices  P,  P',  Q,  R  or 

R'.  We  can  eliminate  this  possibility  by  showing  that  to  follow 

would 

the  P  policy  on  the  P-Q  boundary /drive  us  into  the  Q  region, 

and  that  similarly  to  follow  the  R’  policy  on  the  Q-R*  boundary  would 

drive  us  Into  the  Q  region.  For  we  ali*eady  know  that  the  Q  policy 

\ 

keeps  us  In  the  Q  region;  nence  It  would  follow  that  any  convex 
linear  combination  with  one  of  the  policies  P  or  R'  would  drive  us 
Into  the  Q  region. 
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First,  let  us  consider  the  result  of  following  P  on  the  P-Q 
boundary.  Looking  at  the  differential  equations  (<^.1),  we  see 
that  following  P  rather  than  Q  Increases  Xa  more.  Increases  Xs 
less,  and  consequently  Increases  r  more.  Hence  the  solution 
trajectory  must  go  Into  the  Q  region.  A  similar  argument  gives 
the  same  result  on  the  Q-R'  boundary. 

Finally,  we  note  that  there  Is  also  the  Intermediate  case 
*  1.  which  forms  the  natural  connection  between  the  two 
cases  already  discussed.  As  should  be  expected  the  policies 
followed  are: 

For  r  <  l/Ya  »  l/lfa!  P 
For  r  >  lAa.  *  <  T*:  P' 

For  r  >  l/^a»  T  >  T«:  R' 

In  this  case  the  boundary  between  the  P  and  R'  regions  Is  of 
Interest.  To  follow  P  would  drive  us  Into  the  R'  region,  while 
to  follow  P'  would  drive  us  Into  the  P  region.  But  there  Is  an 
Intermediate  policy  which  keeps  us  on  the  boundary.  The  policy 
Is  given  by 


It  Is  this  policy  which  Is  followed.  This  Illustrates  that  care 
must  be  used  In  assuming  that  an  optimal  policy  consists  only 
of  vertex  choices. 


1.  Bellman 

2.  - 


P-4  92 
^8- 


BIBLIOGRAPHY 


R.,  "Bottleneck  Problems  and  Dynamic  Programming," 
Proc.  Nat.  Acad.  Scl. ,  ?9  (Sept.  1953) »  PP*  9^7-51. 

- ,  "Bottleneck  Problems,  Functional  Equations,  and 

Dynamic  Programming,"  RAND  Paper  No.  P-483  (submitted 
to  Econometrics ) . 


\ 


AFBMD  Library  Charge-Out  Card 

Document  No:  55-alL37  Copy  No:  ^  O/A  Report  No:  ?  **90 

Title:  sac  APPuawoB  cr  the  tobcot  or  dmamic  pfwcK«i*«nc  a  rbvew 


Orig.  Ag'y:  RAKD 


SXCMUBD  SELIJtUI 


Borrower's  Name 


Doc.  Date: 


lit  Mqr 


Date  Out. 


Contract  Na: 


Borrower's  Signature 


